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SYSTEMS AND METHODS FOR GENERATING 
AN APPLIANCE WITH TIE POINTS 

BACKGROUND 

5 This application is related to U.S. U.S. Patent Application Serial No. , 

entitled "Systems and Methods for Integrating 3D Diagnostic Data." The present 
invention relates to three-dimensional (3D) representations of objects such as anatomical 
structures. 

Many applications, including medical and dental applications, rely on data 
10 visualization to provide a holistic display of information and to maximize the amount of 
information that can be conveyed at once. In data visualization, a graphical mapping 
from an information space to a display space is performed. The process allows a user to 
visually explore the 3D representations or models of objects. The 3D models become 
much more useful when they are integrated, that is, when a number of related models are 
15 merged and overlaid to provide spatial context, among others. 

For example, knowledge of the spatial relationships of the jaws, teeth and cranium 
is needed in various dental applications. Such relationships include the relative positions 
of hard structures (teeth and bones) and overlying soft tissues and skin. The customary 
types of physical record obtained by dental clinicians include photographs of the face 
20 (both extra-oral and intra-oral), X-ray images of the skull taken from different 

projections, and plaster study casts of the teeth themselves. Conventionally, most 
diagnoses are currently made using 2D photographs, 2D X-ray films and 3D plaster study 
casts. 



SUMMARY 

A system integrates anatomical information from a plurality of sources of 
information. The system receives two or more three dimensional (3D) anatomical maps 
sharing a common plane specified by three or more marker points common to the two or 

5 more maps; places one or more marker points on one or more teeth; generates a digital 
teeth model with the marker points; and aligns the two or more 3D anatomical maps and 
the teeth model using the marker points. 

Implementations of the system may include one or more of the following. The 
anatomical information can be stereo craniofacial data. One of the anatomical maps is an 

10 X-ray map. The X-ray map is generated using correlated points on X-ray pairs and using 
y-parallax measurements. The X-ray information is stereo. The system can calibrate one 
or more X-ray sources. The calibration determines a principal distance from an X-ray 
source to a film plane. The system can also characterize internal dimensions of the one 
or more X-ray sources by locating an X-ray film relative to an X-ray source. One of the 

15 anatomical map can be a 3D image map. Another anatomical map can be a dental map. 
Each marker can be a tie point. The system can also use discrete anatomical landmark 
information. The system can display the aligned maps as an integrated 3D anatomical 
model. 

In another aspect, a method visualizes anatomical information from a plurality of 
20 sources by receiving X-ray information having X-ray marker information; receiving a 
three-dimensional anatomical information having anatomical marker information; 
receiving a three-dimensional teeth model having teeth marker information; aligning the 
X-ray information, 3D anatomical information, and the 3D teeth model using the marker 



information; and displaying the aligned X-ray information, 3D anatomical information, 

and the 3D teeth model. 

In another aspect, a system includes an appliance with one or more teeth markers 
embedded therein; an X-ray camera receiving X-ray information with X-ray marker 
5 information; a three-dimensional digital camera receiving three-dimensional anatomical 
information with anatomical marker information; a dental scanner to generate a three- 
dimensional teeth model with teeth marker information; a computer to align the X-ray 
information, 3D anatomical information, and the 3D teeth model using the marker 
information. 

10 Advantages of the invention may include one or more of the following. The 

system improves the capability of orthodontic practitioners to develop diagnoses 
orthodontic treatment plans. 

Through the use of markers called tie points, the system allows merging of 
multiple data sources such as X-ray images, study cast images or models, and facial 

15 images into a single unified coordinate system. Because the coordinate frame of 

reference is unified, two or more data sources that are not directly tied together can in 
fact be compared. For example, because the study casts are referred to the X-ray 
anatomic framework, and the facial imagery is tied to the same X-ray anatomic 
framework, therefore the study casts and facial imagery can be compared to each other, 

20 even though they are not directly tied to each other. Overall, the system also provides a 
uniform accurate coordinate system in three dimensions, meaning that accurate unbiased 
measurements can be obtained, regardless of subject orientation. 

The system permits each type of anatomical structure to be imaged using an 
appropriate imaging modality and perspective best suited for locating that structure, after 

25 which all the optimal locations can be merged into a common 3D map by using tie points. 
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The system is robust in construction, simple to operate and requires a minimum departure 
from the technician's usual procedures. The user only needs to identify and locate 
landmarks, after which all measurements, computations, and integrations of data from 
different primary sources are made semi-automatically or automatically by computers 
5 and computer-aided technicians. 

The ability to tie the data sources together opens up new diagnostic and treatment 
planning opportunities. The system supports a dynamic analysis of the functioning of the 
jaws, teeth, and musculature. 

10 BRIEF DESCRIPTION OF THE DRAWINGS 

Fig. 1 is a diagram showing a system for performing integrated three-dimensional 
craniofacial analysis from a combination of data sources 

Fig. 2 shows the sequence of processes by which integrated 3D graphical and 
numerical data are generated for storage, recovery and utilization by the database shown 
15 in Fig. 1. 

Figs. 3 A, 3B and 3C illustrate top view, front view and perspective view of an 
embodiment of a stereo X-ray machine. 

Figs. 4A-4F illustrate an exemplary calibration array. 
Figs. 5 A-5B shows a cassette carrier or holder. 
20 Fig. 6 shows two views of a tie point fitting which a clinician can bond (attach) 

directly to a tooth. 

Fig. 7 shows a type of tie point fitting for use on the face of a patient. 
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Figs. 8A and 8B illustrate radiographic markers that are embedded in an 
appliance. 

Fig. 9 illustrates an exemplary mounting of a number of tie points on a subject's 

face. 

5 Fig. 1 0 is a diagram showing a data management system supporting the 

integration of three-dimensional craniofacial analysis from a combination of data sources. 

Fig. 1 1 is a simplified block diagram of a data processing system used to generate 
an appliance. 

DESCRIPTION 

10 Fig. 1 shows a system that enables digitally encoded graphic representations the 

teeth and the facial surface to be merged with accuracy and precision, making possible 
three dimensional diagnosis, treatment planning, and evaluation of results in orthodontics 
and other types of dental and craniofacial treatment. 

Referring now to Fig. 1, a set of radiopaque and photographically visible spherical 

15 tie points 12 are temporarily mounted on the surfaces of the face and teeth of a patient 1 0. 
With these tie points 12 in position, two or more skull X-ray images are generated from 
different perspectives using an X-ray camera 20. The camera 20 can be a stereo camera 
with two X-ray emitters, or can be a calibrated single emitter system. The X-ray images 
can utilize either coplanar or biplanar geometry. Each tie point 12 is unambiguously 

20 visible on two or more radiographic projections. The spatial relationships between the X- 
ray emitter(s) and the X-ray film or other image-capture medium are also known with 
accuracy and precision. The X-ray camera 20 is calibrated with a specialized calibration 



apparatus and associated software. The X-ray camera 20 uses a cassette-holder installed 
at the opposite end of the X-ray camera from the X-ray tube. 

The skull X-ray images can be of either the bi-planar or co-planar type. If they are 
of the coplanar type, three-dimensional locations of structures not lying in the sagittal 
5 plane can be obtained. Thus, 3D wire frame representations of skull and jaw anatomy can 
be constructed in a manner similar to the 2D tracings of traditional orthodontic 
cephalometrics. 

A 3D image-capture system 30 captures digital 3D images of both 1) the tooth 
surfaces and their associated tie points and 2) the facial surface and its associated tie 

10 points. Particularly, three or more measurable tie points are available in common for 
each pair of 3D images to be integrated or merged. 

Additionally, an appliance 50 is mounted on the patient's teeth. In one of these 
implementations, the tooth tie points are temporarily cemented to the teeth prior to taking 
the impression for the appliance 50. In the other implementation, the impression is taken 

15 first and the tie points are incorporated into the appliance 50 during its fabrication. An 
integration system 70 receives data from the X-ray camera 20, the 3D camera 30 and a 
digital model of the appliance 50 and integrates the data to provide a holistic view of the 
patient 10 for treatment. The integration system 70 is shown in more detail in Fig. 10. 
An exemplary use of the system of Fig. 1 is discussed next. First, the physical 

20 patient records are acquired. During this process, a treating professional such as an 
orthodontist or otherwise secures four to six bonds radiopaque metal spheres to upper 
canines and molars and four radiopaque metal spheres to lower canines and molars. The 
metal markers are called "tie points" such as are shown in Figures 6-8B. They will later 



be used for linking data from images of different types into a common three-dimensional 
coordinate system. High quality upper and lower impressions are then taken, suitable for 
destructive scanning, for example. Additional tie points, like the one in Fig. 7, are 
attached to the facial surface prior to taking X-rays and facial photographs in a 

5 configuration similar to that shown in Fig. 9. Lateral and frontal stereo X-ray images and 
a single panoramic X-ray image are taken with the tie points in place, using a suitable 
digital or analog X-ray system. Frontal and lateral 3D facial photographs are taken using 
a 3D photographic system. Fully digital versions of all the above records are stored in 
the integration system 70. 

10 In order to integrate the data from all sources the positions of the tie points on 

each physical record must be located. On study casts such as those made in accordance 
with US Patent No. 5,975,893 issued to Align Technology, Inc., the 3D data are 
generated by the Align software/hardware system. On the facial photographs, the 
procedure can be done by an operator on screen, also directly in three dimensions using a 

15 suitable 3D camera. The reconstruction of X-ray information on tie point locations must 
be done individually on each image as an on-screen operation. However, it can be done 
by a minimally trained technician or assistant and does not require any orthodontist's 
time. Automated digital pattern recognition methods may also be used to locate tie points 
on images without human intervention. 

20 For the three space coordinates of an anatomical landmark on an X-ray 

cephalogram to be precisely known, the landmark must be independently located on both 
images of an X-ray stereo pair. This step requires involvement of either an orthodontist 
or a trained technician or dental assistant. Since the tie points are in the same position as 
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the face during the taking of both the lateral X-ray images and the frontal X-ray images, 
all anatomical landmarks can be methodically transferred and displayed in the proper 
anatomic relationship to each other. The existence of the tie points allows landmark 
information from lateral images to be precisely related to information from frontal images 

5 despite changes in position of the head between stereo pairs. In addition, when 

information from a panoramic X-ray image is combined with information from the study 
cast map, the positions of tooth roots will be able to be located with only slightly reduced 
accuracy and precision. 

At this point, the treating professional has available a fully explorable multi- 

10 layered representation of the skull in three dimensions. Measurements from the teeth to 
each other and to the surfaces of the face can be performed using a mouse-driven cursor. 
If the treating professional desires conventional two dimensional cephalometric 
information, he or she can obtain it by locating landmarks and surfaces solely on the 
centered lateral and centered frontal images from the emitter 310 without the emitter 312. 

15 In order to obtain full skeletal information in three dimensions, the treating professional 
or a trained assistant will be required to locate the same landmark on each of two stereo 
X-ray images using the emitters 310-312. 

In one implementation, to assist in relating the data from the data sources, 
radiopaque and photographable metal spheres are attached to the surface of the face while 

20 similar spheres are mounted on temporary removable plastic templates which are formed 
on the study casts and are readily transferable between the casts and the mouth. All these 
reference markers are mounted on the face or in the mouth at the time the appropriate 
stereo X-ray pairs are exposed. Following processing of the images, the tie points 
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coordinates are located and mapped in a fashion identical to that employed for any 
anatomical landmark of interest. Landmark locations on the final maps are usually 
expressed in terms of an anatomical coordinate system based on the lateral skull X-ray 
film map. After the various individual coordinate maps for the X-ray films, study casts, 

5 and facial photography have been generated, they are integrated with each other through 
a series of mathematical transformations. After these transformations, the resulting 
integrated map can be visualized, manipulated and measured on a computer monitor. 

The information stored in the integration system 70 is merged to create a holistic 
view of the patient data from a plurality of data sources. Fig. 2 shows a process 199 

10 which uses information from the raw data acquisition module 102 and database 104 of 
Fig. 1. First, an appliance containing radiopaque tie points or markers is created. The tie 
points are identifiable markers that are visible in multiple types of images, or across 
multiple time points within a single type of image. The locations of the markers are 
specified in advance. When digital camera 3D data is merged with stereo X-ray data, 

15 common points in both sets of images are needed. In this case, small radiopaque spheres 
are temporarily placed on the face of the patient. They show up as small "targets" in the 
digital facial images. Because they are radiopaque, they also show up in the X-rays. Then 
3D coordinates are computed for these common tie points from both the stereo X-rays 
and from the digital facial photographs. Once the coordinates are known from both 

20 sources, a mathematical transformation can be applied to rotate and translate the facial 
data into the same frame of reference as the X-ray data. 

In the process 199, the radiopaque markers are then positioned on the patient's 
teeth at their designated locations using the aligner (step 200). At the same time, the 



patient is also fitted with a second series of tie points that consists of radiopaque markers 
on the face. Next, stereo X-ray images are acquired using a calibrated X-ray machine 
(step 202). Lateral and frontal images are digitized, and three dimensional coordinates 
are computed for the tie points on the teeth, the tie points on the face, and any desired 

5 anatomical landmarks visible on any two or more X-ray images (step 208). 

From step 202, 3D facial data are acquired using a 3D camera (step 204). Next, 
3D coordinates for facial Tie Points from facial digital image are computed (step 210). 
From step 204, one or more appliances are created (step 206), and 3D coordinates for tie 
points on teeth computed from the appliances (step 212). The 3D coordinates are 

10 computed from the study casts for the tie points on the teeth. At this point, 3D 

coordinates are available for: a) anatomical features from X-rays; b) tie points on the 
teeth from X-rays; c) tie points on the face from X-rays and facial images; e) tie points on 
the study cast and therefore the model of the dentition. 

During this process, the correspondences between the markers on the teeth and 

15 their images on the X-ray images are identified. Also, the correspondences between the 
markers on the face and the images of the markers on the X-ray images are identified. 
Using the two pairs of correspondences, the three data sets are mathematically registered 
in 3D space as a computer operation so that the corresponding tie points coincide. The 
integrated map can now be viewed in any desired orientation. 

20 Once aligned, the system can display all three data sets together with the upper 

and lower tooth images. The face image is correctly positioned with respect to the X-ray 
image. The stereo X-ray and the resulting anatomical coordinates act as a framework or 
scaffold upon which the other data sources (study cast and facial images) are hung. In this 



way, the study cast data and the facial image data can be visually and analytically 
compared to each other, even though there is no direct data connection between the two. 

3D coordinates of tie points on the patient's teeth and face (from X-ray Images) 
are transformed to an anatomical framework (step 214). Further, 3D Coordinates of Tie 
5 Points on the study casts are transformed to the teeth tie points from the stereo X-rays 
(step 216). 

From step 216, 3D coordinates of tie points from the patient's face (from facial 
digital images) are transformed to the facial tie points (now in the anatomical frame of 
reference) from stereo X-rays (step 220). 

10 In sum, two appliances representative of the two jaws are fabricated upon the 

study casts of the upper and lower dental arches in such a manner that the teeth can 
occlude freely and without interference with the appliances in place. Each appliance has 
fastened to its surface three or more radiopaque metal markers called tie points. The 
same technique is used on the facial 3D photos to define a known plane in the face by the 

15 expedient of placing three or more radiopaque and photographable markers (tie points) on 
its exterior surface. The stereo X-rays of the skull are made with the appliances in place 
in the mouth and with the facial tie points in position. Because of the radiopaque 
property of the tie points, they will also be unambiguously identifiable in the stereo X- 
rays of the skull. 

20 In one operating approach, a clinician bonds tie points to teeth using a fitting such 

as that of Fig. 6. Clinician then takes one or more impressions of the patient during a 
records session. The impressions are used after completion of the records session to 
generate an appliance and coordinates of the teeth and tie points are generated during the 



manufacturing of the appliance. The clinician then takes stereo X-ray images and 3D 
images of the patient. Alternatively, in lieu of the stereo X-ray images, conventional bi- 
planar (frontal and lateral) X-ray images generated from only one emitter can be utilized, 
provided that the X-ray system is calibrated and that the dental and facial tie points are 

5 unambiguously identifiable on the lateral and frontal X-ray images. In such an 

application, however, 3D location of anatomical structures other than those lying in the 
sagittal plane cannot be effected. 

In a second approach, the clinician takes appropriate impressions. An appliance is 
then created with tie points embedded at known locations with respect to the teeth. The 

10 appliance containing the tie points is provided to the clinician who positions it in the 
patient's mouth. X-ray and 3D facial images are then captured, as discussed above. 

Figs. 3 A, 3B and 3C illustrate top view, side view and perspective view of an 
embodiment of a stereo X-ray machine. The stereo X-ray machine has two emitters 310- 
312 that are mounted on a horizontal bar. In this embodiment, the emitter 3 12 is 

15 positioned 18 inches from the emitter 310. The emitter 310 is positioned 60 inches 

directly from a patient 316. The patient 3 1 6 is positioned in a cephalostat (not shown) 
approximately six inches in front of a cassette carrier 318 (Figs. 5A-5B) which defines 
the datum plane. The emitters 310-312 and the cassette carrier are vertically supported 
by a support stand 3 14. The emitter 3 10, the cephalostat and the cassette carrier are 

20 offset from the stand 314 by about twelve inches. The stand 314 allows the X-ray 
equipment to be adjusted to the height of the patient 316. 

This embodiment includes a rigid rectangular structure with twin HRT1 General 
Electric X-ray emitters 310-312 mounted at one end. At the opposite end of the stereo X- 
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ray imaging system, beyond the patient's head is a machined cassette holder defining the 
datum plane 318. The emitters 310-312 and the carrier 3 1 8 taken together constitute the 
"cameras" of the stereo system. The relationship between the cassette holder 318 and 
both X-ray emitters 3 1 0-3 12 is known through previous use of a calibration device. The 

5 calibration process consists of imaging a calibration cage with three planes of precisely 
known radiopaque targets. Using a simultaneous least squares adjustment ("bundle 
adjustment"), the 3D locations of the X-ray emitters are determined. 

The mathematical calculation of the 3D positions of the tie points and anatomical 
structures on the stereo X-ray images depends upon having accurate and precise 

10 information on the physical relationships between the focal spots of the two entities (3 1 0 
and 312) and the surface of the X-ray film or its digital equivalent located in the cassette 
carrier at 3 18. The information needed includes: (1) the distance between the two focal 
spots 310 and 3 12, measured parallel to the surface of the film in the cassette carrier; (2) 
the perpendicular distance between each focal spot and the plane of the film surface, and 

15 (3) the precise location of the cassette within the cassette carrier for each exposure. This 
location differs slightly for different projections as may be seen by examining the X-ray 
intersections 316 and 318 in Fig. 3 a. To obtain this data, a calibrated array, the cassette 
carrier previous mentioned, and an "auxiliary calibration checking frame." 

Figs. 4A-4D illustrate an exemplary calibration array 500. An imageable 

20 structure of known dimensions is placed in a location from which it can be X-rayed. The 
structure consists of a radiolucent framework upon which are mounted a number of 
radiopaque points whose three space locations with respect to each other are known with 
accuracy and precision. In the simplified illustrated case there are four perpendicular 
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radiolucent plastic rods 502, 504, 506 and 508 at the top and bottom of which the 
radiopaque points are mounted perpendicular to each other. When the X-ray image is 
exposed, the shadow points 503, 505, 507 and 509 of the top of the rods 502, 504, 506 
and 508 will be cast upon the film surface, radially displaced with respect to the original 

5 three space position of the rod top point. A line drawn in three space which passes 
through both any rod top point and the image of that point on the X-ray film will also 
pass back along the path of the ray which erected the image through the focal spot from 
which the ray originated. This principle is shown in photogrammetry as the principle of 
resection. Since the same principle applies to all four rods, a series of lines passing 

10 through an X-ray focal spot 510 (corresponding to point 310 or 312 of Fig. 3A) may be 
generated, the intersection of any two of which would identify the focal spot uniquely if 
the measurements were without error. Since there is always error, redundancy is added in 
this implementation by the use of four rods rather than two. 

By repeating exposures for both X-ray sources, (the equivalents of 3 1 0 and 312), 

15 the distance from each X-ray source to the film plane can be computed. Next, a point 
520 on the film plane at which the perpendicular ray strikes the point is identified. This 
point is called the "principle point" and the distance between the principle points of both 
X-ray images in the precise measurement of the distance between the focal spots of the 
two X-ray sources measured parallel to the film surface. 

20 To locate the principle point for each X-ray source, a line is drawn in the plane of 

the film passing through the images of each rod top point and its rod own bottom point. 
Information from two rods can be used in the absence of measurement error and the 
redundant information arising from the use o four rods strengthens the calculations in the 



presence of error. At this point, the distance from each X-ray source to the film plane, 
the point of contacts on the film plane of the principle ray from each source, and the 
distance between the two X-ray sources are known. 

In certain cases, it is impractical to use the system of four perpendicular rods 

5 surrounding the patient's head because of physical clashes with the cephalostat that is 
required for the control of head orientation in object space. For this reason, another 
exemplary calibration device 518 is shown in Figs. 4E and 4F. The second embodiment 
consists of 3 plastic planes 520-524 assembled in parallel layers, successively separated 
from each other by intervals of a known value, in one embodiment 9.5 inches. This 

10 device is positioned between the X-ray sources and the film plane, as near as possible to 
it. In this embodiment, the plane of the device nearest the film plane has nine radiopaque 
markers in carefully measured locations; the other two planes have four markers each. 
While the principle of measurement is still resection, the redundance of this calibration 
device avoids the need to orient it with great precision with respect to the film plane. The 

15 approximate precision in measuring the principal distance with this apparatus is in the 
one to three millimeter range. The distance between the X-ray emitters can be measured 
with one millimeter. 

The next task is to obtain precise information about the precise orientation of the 
film cassette with respect to X-ray emitters. For this purpose, a cassette holder 318 (Figs. 
20 5 A-5B) is rigidly attached to X-ray emitters or sources 310 and 312. The holder 318 has 
on its surface a set of precisely positioned radiopaque fiducial points (324 being typical) 
whose precise coordinates are known through direct measurements. These fiducials, in 



sets of at least four are imaged on the film surface during exposure, thus identifying the 

precise position of each film with respect to its X-ray source. 

Since the control array cannot be maintained in position all the time, a slightly 

lower order auxiliary calibration checking frame is supplied. This device (not shown) 
5 consists of a smaller number of radiopaque spheres mounted on a plastic cone or box in 

known positions which can be periodically fastened to a cassette positioned within the 

cassette holder. While it is not sufficiently redundant for use as a calibration array, it is 

powerful enough when used periodically to check the robustness of calibration achieved 

using the full calibration array. 
10 The sequence of steps in the operation of the stereoscopic X-ray system is as 

follows: 

1 . The subject's head is positioned within the object space, secured in a head holder 
or cephalostat to minimize movement between exposures. 

2. A cassette containing an unexposed X-ray imaging medium (film or digital 
15 medium) is positioned in the cassette carrier 318. 

3. The X-ray emitter located at a first station is fired, exposing the film or a digital 
image capture medium. 

4. The first cassette is removed and replaced with a new cassette containing a new 
unexposed film or digital image capture screen medium. 

20 5. The X-ray emitter located at a second station is fired exposing the film. 

After producing the stereopair of lateral skull films, the patient and the 
cephalostat are next rotated 90 degrees such that the patient faces the cassette carrier 318. 
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Two more films are exposed from the first and second stations. These constitute a frontal 
X-ray stereopair. The films pairs are processed and examined by a monocular analytic 
method in which a person examines the two images sequentially and marks the locations 
of the cassette holder fiducial points, the tie points, and one or more sets of anatomical 
5 landmarks. The location of these structures on the X-ray images can be done manually or 
by using suitable computer programs. 

Software is then used to examine the tracings for each film as a unit of 
information. If duplicate tracings have been made, the software rotates, translates and 
rescales the multiple tracings for each X-ray image to a least squares best fit upon the 
10 directly measured known X and Y coordinates of the fiducial points on the datum plane. 
The set of coordinate values for all landmarks of each film is stored as a block. The 
coordinates of the principal point of each film and of the fiducial points on the datum 
plane are added to the set of coordinate values for the X-ray image. 

The coordinate files for the two films of each stereopair are now used to produce 
15 a three dimensional map in the following manner. The file for the film from second 

camera station 312 (the offset film) is rotated and translated such that its four registration 
points are best fit upon the registration points of the film from the first camera station 3 1 0 
(the centered film). The X parallax of each point in the system may now readily be 
computed as the AX between any landmark on the "centered" film and its conjugate on 
20 the "offset" film, and the altitude of the landmark is computed automatically using 

standard photogrammetric equations. In addition, the Y parallax for each landmark and 
registration point is computed as measure of the degree to which the same physical 
structure was actually identified on both films of the stereopair. More information on the 
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transformations is described in Elements of Photogrammetry, Paul R Wolf, McGraw-Hill 
(1974). Systematic Y parallax deviation for anatomical landmarks which are 
significantly greater than those for the fiducials are indicators of patient movement 
between the exposure of the two images of the stereopair. 

5 In one embodiment, the raw data acquisition module 102 receives 3D facial data 

from a 3D camera such as the Venus3D camera available from 3DMetrics of Petaluma, 
California. The 3DMetrics camera uses a 3D Flash light projector and a conventional 
digital camera. The 3D Flash light projector projects a color-coded white light pattern 
onto an object, the digital camera takes an image of the object, then the data is 

10 transmitted to a computer which performs a cross-talk-free-color-decoding operation. The 
2D data is then converted into a 3D image. In this manner, the 3D camera or imaging 
system enables a single regular digital camera to take a 3D image with only one shot and 
does not require any additional mechanical scanning, laser light source, or additional 
shots. The 3D data generated by the 3D camera is then scanned for the location of tie 

15 points. In one embodiment, the tie points are manually identified by a person. Other 
embodiments would provide automatic detection of the tie points. 

Referring now to Figs. 5A-5B, a cassette carrier 318 is shown. The cassette 
carrier has a plurality of tabs or fiducial markers 324 and a plurality of stop points 322. 
An X-ray cassette is inserted into the cassette carrier and slid into position behind the tabs 

20 324. The tabs 324 are radiolucent and carry radiopaque fiducial control points similar in 
size to the tie points. However, they differ from tie points in that their exact spatial 
location with respect to the X-ray tubes 310 and 3 12 are previously shown through prior 



calibration. The stop points 322 are adjustable to limit the lateral traversal of the cassette 
at various desired positions. 

Fig. 6 shows two views of a tie point fitting used with for use with a direct 
bonding onto the teeth of a patient. The tie point fitting has a plastic carrier 330. 
5 Mounted on top of the plastic carrier 330 is a radio opaque tie point 332. As shown in the 
side view and the edge view, the plastic carrier 330 has serrations at a bottom end to 
facilitate mounting of the tie points 332 onto a tooth of a patient using a suitable glue or 
bonding system. 

Fig. 7 shows a tie point fitting for use on the face of a patient. A tie point 342 is 
10 positioned on top of a paper or adhesive tape carrier 340. The carrier 340 has an adhesive 
backing that attaches the tie point to the face of the patient. 

Next, appliances with radiographic markers will be discussed. In one 
implementation, radiographic markers are embedded in an appliance 300, shown in Figs. 
8A and 8B. The appliance 300 is a polymeric shell having a cavity shaped to receive 
] 5 teeth. The polymeric shell typically fits over all teeth present in the upper or lower jaw. 
The polymeric appliance 300 is preferably formed from a thin sheet of a suitable 
elastomeric polymeric, such as Tru-Tain 0.03 in. thermal forming dental material, 
marketed by Tru-Tain Plastics, Rochester, Minnesota 55902. The appliance 300 has four 
radiopaque metal pins 302, 304, 306 and 308. In one embodiment, the radiographic 
20 markers or tie points are stainless steel spheres that are 3/32" in diameter, grade 100 
material with sphericity within 0.0001 inch and with a hardness rating of Rockwell C 
39. Only three tie points are sufficient to determine a common plane for merging two 3D 
maps if they are located perfectly. However some redundancy is desirable and so the 
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present system employs a minimum of four. Although tie points are shown in Figs. 8A- 
8B, six tie points used in the event that the patient has fillings that way obscure some of 
the tie points in the X-ray images. 

One procedure for creating the appliance 300 with the radiographic tie points 302- 

5 308 is discussed next. First, a dental professional takes one or more PVS impressions 
and a wax bite for measuring inter-arch relationship. The impressions are used to 
generate one or more models, which are then scanned. The dental professional then sets 
up a bite registration of both arches, and tooth-attachments are mounted onto the desired 
teeth which is exactly same size and shape of the actual tie points 302-308. In one 

10 embodiment, the attachments are mounted on three teeth in each quadrant (canines, first 
molars and second molars for a total of twelve attachments in both arches. Aligners are 
then fabricated with sufficient spaces for retaining the tie points 302-308. The tie points 
302-308 are positioned onto the spaces, and a thin film of unfilled composite adhesive is 
applied to secure the tie points 302-308. Next, the dental professional opens the occlusal 

15 surface of aligners to establish contact between the opposing maxilla and mandible casts. 
Further, the dental professional can polish the appliance 300 to make it more comfortable 
to wear. 

Fig. 9 illustrates an exemplary mounting of the tie points on a face 350. 
Points 351 and 352 are located on the forehead above the eyebrows and are called 
20 supercilliary points. Point 353 and 354 are the zygomatic points and are located on the 
anterior aspect of the cheekbone. Points 355 and 356 are the gonial tie points and are 
located at the angle of the lower jaw. Point 357 is the nasal tie point and is located on the 
tip of the nose. Points 359 and 360 are the anterior jaw tie points one being located on 



each side of the chin. Points 362 and 364 are the ear tie points located just above the 
earlobe. In general, at least 3 tie points must be visible on at least two images of any set 
of averages to be used for merging map components. Further, the tie points should 
surround the biological region of interest so that measurement to the biological point can 

5 be interpolated. Additionally, the geometry is more efficient if no more than 2 points are 
situated along a common line. 

In order to avoid the possibility that images of the tie points on the right and the 
left sides of the face are confused when locating them on lateral X-ray images, the facial 
tie points on the side of the face nearer to the X-ray emitters 3 10 and 312 are slightly 

10 larger and are positioned slightly lower on the face than are the facial tie points on the 
side of the face nearer to the cassette carrier 318. Similarly, to avoid errors, the tie points 
on the upper teeth are a slightly different size from the tie points on the lower teeth. The 
information in the tie points alone is itself sufficient for the construction of an integrated 
three-dimensional craniofacial image map without any specialist intervention or time 

1 5 commitment. 

In another implementation, the procedures for the 3D mapping operation are as 
follows: 

(1) Conventional upper and lower impressions are taken and conventional dental 
casts are poured and trimmed. 
20 (2) Without damaging the casts, thin cold cure acrylic upper and lower appliances are 
fabricated in such a way as to be stably positionable in the mouth without occlusal 
interference. 



(3) Three or more radiopaque tie points are imbedded into each acrylic appliance with 
the top points visible at the surface. These are the cast control tie points. (Note: 
steps 4-7 are performed during the same visit.) 

(4) The acrylic appliances are placed in the subject's mouth. 

5 (5) Three or more radiopaque tie points are fixed to the lateral and frontal aspects of 
the patient's face. These are the facial control tie points. 
(6) The patient is positioned in the object space of the stereo X-ray system as 
described above and the lateral and frontal X-ray stereopairs are taken as 
described. 

10 (7) The patient is positioned in the object space of the facial photographic system and 
lateral and frontal photographic stereopairs are taken. 

(8) The upper and lower acrylic appliances are returned to their respective casts and 
the casts are individually stereophotographed in appropriate positions. 

(9) (Optional) If checks on bite relationship are desired, the appliances may be 
15 returned to the mouth and a wax check bite registration may be made. The 

appliances and the wax bite are removed from the mouth, fastened together and 
stereo X-rayed. The relationship of the tie points on this X-ray pair constitutes a 
mathematically applicable control on the correctness of the bite relationship 
during the taking of the lateral and frontal stereo X-ray pairs. 
20 (10) Appropriate tracings are made of the X-ray and photographic stereopair. Note 
that each tie point must be located both on the photographs and on the X-ray 
images. 

(11) The tracing data is reduced to coordinate form. 
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(12) The two dimensional coordinates of all points used to produce a single integrated 
dimensional craniofacial map. 

(13) Stored maps from different time points or different subjects may be compared as 
desired. 

5 

Additional supplementations of the stereo X-ray system have been used in 
prosthetic dentistry (R. Bellagamba, FR Brigante, S. Baumrind (1986) 1 Prosthetic 
Dent 55:625-28) and in orthopedic surgery for the detection of pseudarthrosis following 
lumbar fusions (N. Chafetz, S. Baumrind, JE Morris, HK Genant, EL Korn (1985) Spine 
10 10-368-75) and for the detection of loosening of femoral prostheses (N. Chafetz, S. 
Baumrind, W. Murray, HK Genant, EL Korn (1985) Clinical Orthopedics and Related 
Research 210-60-67). 

Fig. 10 shows a data storage and management system for processing integrated 
three-dimensional craniofacial analysis from a combination of data sources. The system 
15 integrates 3D data from stereo cranial X-rays, stereo digital facial images, and digital 
models of the dentition. The system includes a raw data acquisition module 102. This 
system acquires patient data including data from headfilm scanning, stereo images, 
patient data, digital facial images, study cast images, X-ray images and video, collateral 
material associated with the patient, and treatment records. The output of the raw data 
20 acquisition module 102 is provided to a database 104. 

The database 104 maintains 2D and 3D coordinates of patient data, computed 
measures derived from the patient data, digital facial images, study cast images, X-ray 
images and video, collateral material associated with the patient, and treatment records. 
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The database 104 also receives data from a film digitizing module 112, which extracts 2D 
and 3D coordinate information from the raw images of the patient. The database 104 can 
also output data using a publication module 114. The publication module 1 14 can 
publish image as well as text data for a variety of applications, including teaching and 
5 data sharing. 

The database 104 can be accessed using a variety of browsers 106, 108 and 1 10. 
The browser client 106 can view images and data. The browser client 106 can overlay 
landmarks upon request, and perform various analytical operations. The analysis client 
108 performs spatial and temporal analysis and other statistical operations, among others. 

10 The client 108 contains one or more analytical software tools for analyzing data in the 
database 104. The training client 110 performs tracer analysis and displays ellipses, 
among others. The client 1 10 is a training module for training and calibrating human 
judges so that they are consistent in identifying points on images. The display ellipses 
store information relating to acceptable deviations between two judges. As such, the 

15 display ellipses improve the quality of the landmark locations on X-ray images and 
photographs. Thus, the database 104 supports training (calibrating) clinicians and the 
treating professionals in identifying three-dimensional landmark locations, and 
integrating information from pairs of X-ray images to yield three-dimensional data and 
comparing information from multiple time points. 

20 Fig. 1 1 is a simplified block diagram of a data processing system 600 that may be 

used to generate the appliance 300 of Figs. 3A-3B. The data processing system 600 
typically includes at least one processor 602 that communicates with a number of 
peripheral devices via bus subsystem 604. These peripheral devices typically include a 
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storage subsystem 606 (memory subsystem 608 and file storage subsystem 614), a set of 
user interface input and output devices 618, and an interface to outside networks 616, 
including the public switched telephone network. This interface is shown schematically 
as "Modems and Network Interface" block 616, and is coupled to corresponding interface 

5 devices in other data processing systems via communication network interface 624. Data 
processing system 600 could be a terminal or a low-end personal computer or a high-end 
personal computer, workstation or mainframe. 

The user interface input devices typically include a keyboard and may further 
include a pointing device and a scanner. The pointing device may be an indirect pointing 

10 device such as a mouse, trackball, touchpad, or graphics tablet, or a direct pointing device 
such as a touchscreen incorporated into the display, or a three dimensional pointing 
device, such as the gyroscopic pointing device described in U.S. Patent 5,440,326, other 
types of user interface input devices, such as voice recognition systems, can also be used. 
User interface output devices typically include a printer and a display subsystem, which 

15 includes a display controller and a display device coupled to the controller. The display 
device may be a cathode ray tube (CRT), a flat-panel device such as a liquid crystal 
display (LCD), or a projection device. The display subsystem may also provide non- 
visual display such as audio output. 

Storage subsystem 606 maintains the basic required programming and data 

20 constructs. The program modules discussed above are typically stored in storage 

subsystem 606. Storage subsystem 606 typically comprises memory subsystem 308 and 
file storage subsystem 614. 
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Memory subsystem 608 typically includes a number of memories including a 
main random access memory (RAM) 610 for storage of instructions and data during 
program execution and a read only memory (ROM) 612 in which fixed instructions are 
stored. In the case of Macintosh-compatible personal computers the ROM would include 

5 portions of the operating system; in the case of IBM-compatible personal computers, this 
would include the BIOS (basic input/output system). 

File storage subsystem 614 provides persistent (non- volatile) storage for program 
and data files, and typically includes at least one hard disk drive and at least one floppy 
disk drive (with associated removable media). There may also be other devices such as a 

10 CD-ROM drive and optical drives (all with their associated removable media). 

Additionally, the system may include drives of the type with removable media cartridges. 
The removable media cartridges may, for example be hard disk cartridges, such as those 
marketed by Syquest and others, and flexible disk cartridges, such as those marketed by 
Iomega. One or more of the drives may be located at a remote location, such as in a 

1 5 server on a local area network or at a site on the Internet's World Wide Web. 

In this context, the term "bus subsystem" is used generically so as to include any 
mechanism for letting the various components and subsystems communicate with each 
other as intended. With the exception of the input devices and the display, the other 
components need not be at the same physical location. Thus, for example, portions of the 

20 file storage system could be connected via various local-area or wide-area network 

media, including telephone lines. Similarly, the input devices and display need not be at 
the same location as the processor, although it is anticipated that personal computers and 
workstations typically will be used. 
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Bus subsystem 604 is shown schematically as a single bus, but a typical system 
has a number of buses such as a local bus and one or more expansion buses (e.g., ADB, 
SCSI, ISA, EISA, MCA, NuBus, or PCI), as well as serial and parallel ports. Network 
connections are usually established through a device such as a network adapter on one of 

5 these expansion buses or a modem on a serial port. The client computer may be a 
desktop system or a portable system. 

Scanner 620 is responsible for scanning casts of the patient's teeth obtained either 
from the patient or from an orthodontist and providing the scanned digital data set 
information to data processing system 600 for further processing. In a distributed 

10 environment, scanner 620 may be located at a remote location and communicate scanned 
digital data set information to data processing system 600 via network interface 624. 
Fabrication machine 622 fabricates dental appliances based on intermediate and final data 
set information received from data processing system 600. In a distributed environment, 
fabrication machine 622 may be located at a remote location and receive data set 

1 5 information from data processing system 600 via network interface 624. 

The invention has been described in terms of particular embodiments. Other 
embodiments are within the scope of the following claims. For example, the three- 
dimensional scanning techniques described above may be used to analyze material 
characteristics, such as shrinkage and expansion, of the materials that form the tooth 

20 castings and the aligners. Also, the 3D tooth models and the graphical interface 

described above may be used to assist clinicians that treat patients with conventional 
braces or other conventional orthodontic appliances, in which case the constraints applied 
to tooth movement would be modified accordingly. 
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What is claimed is 

1 . A method for integrating anatomical information from a plurality of sources of 
information, comprising: 

receiving two or more three dimensional (3D) anatomical maps sharing a 
5 common plane specified by three or more marker points common to the two or more 
maps; 

placing one or more marker points on one or more teeth; 
generating a digital model of the teeth with the marker points; and 
aligning the two or more 3D anatomical maps and the digital teeth model using 
1 0 the marker points. 

2. The method of claim 1, wherein the anatomical information is stereo craniofacial 
data. 

15 3. The method of claim 1, wherein one of the anatomical map is an X-ray map. 

4. The method of claim 3, wherein the X-ray map is generated using correlated 
points on X-ray pairs and using y-parallax measurements. 

20 5. The method of claim 3, wherein the X-ray information is stereo. 

6. The method of claim 3, further comprising calibrating one or more X-ray sources. 

7. The method of claim 6, further comprising determining a principal distance from 
25 an X-ray source to a film plane. 

8. The method of claim 6, further comprising characterizing internal dimensions of 
the one or more X-ray sources by locating an X-ray film relative to an X-ray source. 

30 9. The method of claim 1 , wherein one of the anatomical map is a 3D image map. 
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1 0. The method of claim 1 , wherein the placing of the markers further comprises 
wearing a dental appliance with one or more marker points. 

1 1 . The method of claim 1 , wherein each marker is a tie point. 

5 

12. The method of claim 1, wherein the aligning uses discrete anatomical landmark 
information. 

1 3 . The method of claim 1 , further comprising displaying the aligned maps as an 
10 integrated 3D anatomical model. 

14. A method for visualizing anatomical information from a plurality of sources, 
comprising: 

wearing a dental appliance with one or more teeth markers; 
15 receiving X-ray information having X-ray marker information; 

receiving a three-dimensional anatomical information having anatomical marker 
information; 

aligning the X-ray information, 3D anatomical information, and the 3D teeth 
model using the marker information; and 
20 displaying the aligned X-ray information, 3D anatomical information, and the 3D 

teeth model. 

15. A system, comprising: 

an appliance with one or more teeth markers embedded therein; 
25 an X-ray camera receiving X-ray information with X-ray marker information; 

a three-dimensional digital camera receiving three-dimensional anatomical 
information with anatomical marker information; 

a dental scanner to generate a three-dimensional teeth model with teeth marker 
information; 

30 a computer to align the X-ray information, 3D anatomical information, and the 3D 

teeth model using the marker information. 



29 



1 6. The system of claim 15, wherein the X-ray information is stereo. 

17. The system of claim 15, further comprising a calibration array to calibrate the X- 
5 ray camera. 

18. The system of claim 15, further comprising an X-ray cassette carrier. 

1 9. The system of claim 1 5, wherein each marker is a tie point. 

10 

20. The system of claim 15, wherein the appliance comprises a polymeric shell 
having cavities and wherein the cavities of the shell have different geometries shaped to 
receive teeth. 
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ABSTRACT 

Systems and methods are described for integrating anatomical information from a 
plurality of sources of information. The system receives two or more three dimensional 
(3D) anatomical maps sharing a common plane specified by three or more marker points 
common to the two or more maps; places one or more marker points on one or more 
teeth; generates a digital teeth model with the marker points; and aligns the two or more 
3D anatomical maps and the teeth model using the marker points. 
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FIG. 1 



Radiopaque Tie Points 
affixed to patient face and 
teeth. 
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Stereo X-Ray images 
acquired using 
calibrated CRIL X- 
Ray machine. 



202 



Facial image acquired 
using CRIL 3D 
camera 
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Lateral and frontal images are 
digitized, and 3D coordinates 
computed for Tie Points on teeth, 
Anatomic Landmarks, and Tie Points 
on face, using CRIL software. 



3D coordinates for facial Tie Points 
from facial digital image computed 
using 3D camera software 
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Generate cast of teeth 
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3D coordinates for Tie Points on 
teeth computed from casts 
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3D Coordinates of Tie Points on Teeth and 
Face (from X-Ray Images) transformed to 
Anatomic Framework, using 3D Anatomic 
Landmarks from X-Rays 


^214 




r 


216 

j 






3D Coordinates of Tie Points on Study 
Casts (from Align) transformed to Tie 
Points on teeth (now in Anatomic System) 
from stereo X-Rays 


Study Cast 3D data transformed to 
* Anatomic Frame of Reference 
using teeth visualization software 
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3D Coordinates of Tie Points from Face 
(from facial digital image) transformed to 
Facial Tie Points (now in Anatomic 
System) from stereo X-Rays 
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Facial Imagery transformed to 
* Anatomic Frame of Reference 
using teeth visualization software 
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SCHEMATIC OF X-RAY GEOMETRY 
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